Skip to content

Conversation

@fgiudici
Copy link

@fgiudici fgiudici commented Oct 2, 2020

The storage cache is not completely cleaned when removing images.
Remove also the entries referencing layers by compressed and uncompressed sums.

@mtrmac
Copy link
Collaborator

mtrmac commented Oct 2, 2020

FWIW the main branch fix is a bit more involved: #665 .

@fgiudici
Copy link
Author

fgiudici commented Oct 2, 2020

FWIW the main branch fix is a bit more involved: #665 .

It removes the specific layer id from the layer ids array of the r.bycompressedsum[digest] (and the same for the uncompressed map).
When the last layer is removed, the entry is left in the map with an empty array (but it doesn't harm).
I'm not able to figure out a case in which we can have multiple layer ids referenced by the same compressed digest here... anyone can help me to better understand? :-)

For the record, also backporting #665 would fix the issue mentioned in containers/image#1051.

@mtrmac
Copy link
Collaborator

mtrmac commented Oct 2, 2020

I'm not able to figure out a case in which we can have multiple layer ids referenced by the same compressed digest here... anyone can help me to better understand? :-)

One case, probably reachable through c/image/storage, would be using the same layer on top of different parent layers.

The caller also might not be c/image/storage; e.g. the caller can not supply an ID, causing a random ID to be used, or perhaps the caller might intentionally create a different layer from the same CompressedDigest (e.g. a layer with different UID/GID maps). But I’m getting very much out of my depth WRT possible/valid/expected uses of c/storage, I’ll defer to @nalind.

@nalind
Copy link
Member

nalind commented Oct 2, 2020

I'm not able to figure out a case in which we can have multiple layer ids referenced by the same compressed digest here... anyone can help me to better understand? :-)

One case, probably reachable through c/image/storage, would be using the same layer on top of different parent layers.

The caller also might not be c/image/storage; e.g. the caller can not supply an ID, causing a random ID to be used, or perhaps the caller might intentionally create a different layer from the same CompressedDigest (e.g. a layer with different UID/GID maps). But I’m getting very much out of my depth WRT possible/valid/expected uses of c/storage, I’ll defer to @nalind.

That's right - I find it useful to compare it to source control systems like git, where the same patch can be applied to different branches after they've diverged.

@fgiudici
Copy link
Author

fgiudici commented Oct 5, 2020

Thanks for explaining!
So, makes much more sense to backport #665 here.

@rhatdan
Copy link
Member

rhatdan commented Oct 8, 2020

I think this PR needs to be rebased.

@fgiudici
Copy link
Author

fgiudici commented Oct 9, 2020

I think this PR needs to be rebased.

Rebased, thanks!

Signed-off-by: zvier <[email protected]>
(cherry picked from commit 5dcc8ed)

Signed-off-by: Francesco Giudici <[email protected]>
@rhatdan
Copy link
Member

rhatdan commented Oct 9, 2020

LGTM

@rhatdan
Copy link
Member

rhatdan commented Oct 9, 2020

if layer.MountPoint != "" {
delete(r.bymount, layer.MountPoint)
}
r.deleteInDigestMap(id)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably outside of the scope of this PR, but is there any advantage to backport the deleteInternal() function from master where this particular function is called from there?

Copy link
Author

@fgiudici fgiudici Oct 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. Tried to look into it...
AFAICS the deleteInternal() just contains the core deletion operations once in Delete().
The main difference I can see in commit bcedb54d05 on master (which introduced the function) is that now the Load() function calls deleteInternal() in place of the full Delete() function.
This skips from the Load() the operations left in Delete(), which are:

  • the checks if the layers are mounted
  • the call to the Save() function

As bcedb54d05 commit message says, this was introduced to skip the mountpoints locking in the Save() function, as locking already happens in Load().
Looking at the code in the 1.11 branch, the locking management seems quite different. In particular the Save() function doesn't perform any lock.
So, my guess is that we wouldn't benefit from the deleteInternal() function here (moreover, backporting wouldn't be trivial).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking a look @fgiudici. I was afraid that once you undid one string, too much would unravel with that.

@nalind
Copy link
Member

nalind commented Oct 12, 2020

LGTM, merging.

@nalind nalind merged commit 4d66b2a into containers:release-1.11 Oct 12, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants